fix(forge,ai): clean machine-readable output — forge.error event, no litellm banner, one contract event - #858

Description

This PR cleans up forge's machine-readable output by fixing three defects. All three are older than #850 to #854. There is no issue; the #854 session found the duplicate event.

1. fluid forge --agent reports a failure as a forge.error event, not a traceback (fix(forge)).

• Before: a typed AI error such as copilot_no_llm_provider printed a full Python traceback on stderr. _create_project_minimal logged it with logger.exception, and the CLI's console handler prints that at the default INFO level, in every mode (--agent sets up no logging of its own). The stream said only forge.done with exit_code 1.
• log_forge_failure (cli/_forge_mode_errors.py) handles a FLUID error: the CLIError family, FluidError, FluidUserError, or a missing extra.
  • It logs the error once at ERROR, without the traceback, and logs the traceback at DEBUG.
  • When the caller prints the message itself, the ERROR record is marked fluid_console_shown, so it reaches --log-file but not the console.
  • A crash keeps logger.exception: its traceback is the bug report.
• The same helper notes the failure in a ContextVar. forge.run sends it as one forge.error event before forge.done.
  • The fields are code, message, suggestions and, when the error catalog has one, docs_url. This is the shape of fluid ai test --json's error object.
  • The text is redacted with _console._redact_error_str, because the events go straight to stdout and skip the logging redactor.
• Some failures carry less, because MCP forge_run hands these events to the IDE's model (the copilot tools' typed-error rule, SECURITY_REVIEW I7).
  • An unexpected exception is named by its class only (code: forge_unexpected_error, exception: <Class>).
  • A failed --seed-from sends the console's own words, never the exception's text, which can carry a remote seed's body. Its next step now names what --seed-from accepts; it used to point at a --verbose flag that does not exist.
• MCP forge_run throws away forge's console output, so it now writes each forge.error to the server's log (%r, one line).
• The helper is wired into template mode, the agent loop, the minimal path, ForgeModeErrorBoundary.__exit__ and _run_main's last handler.
  • That last handler now prints a CLIError's or FluidUserError's message and next steps, where it used to print Forge failed: <event code> and a traceback.
  • _template_mode._is_missing_optional is removed; the helper covers its case.
• The --agent help now lists the events the stream really sends. forge.cost was listed but never sent; forge.error, forge.file_written and forge.plan were missing.
• Under --agent, stderr keeps the usual message and suggestion bullets (the founder's choice).
• Some failures still send only forge.done, by design: a usage error (exit 2: the --prompt-profile, --prompt-overlay and --resume flags), and a stop without an exception (blank mode finding a contract in place, a template not found, a rejected composition).

2. fluid ai test --json prints only its report when the call fails (fix(ai)).

• litellm printed its "Give Feedback / Get Help" banner with print(), ahead of the JSON.
• quiet_litellm() sets litellm's own switch, suppress_debug_info, at every forge call into litellm.completion:
  • _get_litellm
  • ai_setup._validate_api_key
  • the agent loop's tool call
  • the agent loop's summarizer
• litellm's Router and proxy set the same switch (litellm issue 5942). The switch gates exactly two print sites in litellm 1.104.0 and 1.109.1: the banner, and a "Provider List" link whose URL is also in the exception text.
• Human fluid ai test output on a failed call differs from main only by the 4 banner lines.

3. fluid forge --agent --blank reports its contract once (fix(forge)).

• Blank mode emitted its own forge.contract_written, and the receipt emitted another for the same file: first by its absolute path, then by its path from the current directory.
• Since fix(ai): #850 follow-ups — target-dir event, one Ollama reader for any host, no gemini fallback #852 the receipt scans every --target-dir, so blank mode's own event goes, as in every other mode.

Design note.

• Who reads what:
  • An IDE agent or CI script shell-runs fluid forge --agent and parses stdout line by line.
  • A human reads stderr.
  • MCP forge_run hands the events to the IDE's model.
• Success: forge.start, then one forge.contract_written per file, then forge.done with exit_code 0.
• Error: forge.error {code, message, suggestions, docs_url}, then forge.done with a non-zero exit_code.
  • The codes are the stable error-catalog events, and the error object has the same shape as in ai test --json.
  • Crashes carry their class only.
• Partial: a stop without an exception sends no invented reason.
• stderr: the CLI's usual message and next steps. A traceback appears only with --debug or FLUID_LOG_LEVEL=DEBUG; a crash keeps its traceback.
• Naming: the event names follow the existing forge.<noun> vocabulary.

Overlap: the "Provider selection" branch also touches fluid_build/cli/forge.py and fluid_build/cli/ai_setup*.py.

• This PR's edits there:
  • forge.py: run (and two new helpers below it), _run_main's last except, _run_blank_mode, and the --agent help string;
  • ai_setup.py: two lines in _validate_api_key.
• Whichever lands second merges main in again.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would break existing functionality)
• Documentation update
• Refactor / code cleanup
• New provider or provider enhancement

Documentation

• Docs PR linked:
  • Docs PR:
• No docs needed — this change does not affect user-facing behaviour
  • Justification:
• Docs TODO — I know docs are needed and will create a PR in forge_docs before this is merged

Pages to update in forge_docs:

• docs/cli/forge.md (Headless agent mode):
  • the event list;
  • forge.error: its fields, its place before forge.done, and which failures send none;
  • tracebacks only at DEBUG;
  • blank mode's single forge.contract_written, reported by its path from the current directory, or by its absolute path outside it.
• docs/cli/ai.md: on a failed call, fluid ai test --json stdout is the report alone, with no litellm banner.
• docs/cli/mcp.md: forge_run results can carry a forge.error event, and a failure is also logged on the server.
• advanced/production-troubleshooting: --debug or FLUID_LOG_LEVEL=DEBUG shows a FLUID error's traceback.
• upgrading: a short "on main" note for the three behaviour changes.

Checklist

• I have read the Contributing Guide
• My code follows the project's coding standards
• I have added tests that prove my fix/feature works
• All new and existing tests pass (pytest)
• I have run ruff check and black with no errors
• New maintained Python files include license headers (python scripts/add_license_headers.py; examples/** exempt)

Testing

Borrow receipts.

• Checked locally:
  • ai test --json's error object;
  • the fluid_console_shown console filter (cli/__init__.py, build_runners/_lineage.py);
  • the SECURITY_REVIEW I7 typed-tool-error rule;
  • _redact_error_str and the _error_catalog docs URLs;
  • the installed litellm's two suppress_debug_info gates and its Router.
• Searched the web for:
  • litellm suppress_debug_info: dspy's disable_litellm_logging sets the same switch;
  • Codex exec --json: an error event plus a turn.failed terminal event;
  • Gemini CLI stream-json: error events plus a final result with a status.
• Reuse: depend on litellm's own switch; follow the agent-CLI error-event pattern.

Tests that fail on main and pass here. 30 new or extended tests, checked on a detached origin/main worktree with only the test files copied in.

• tests/cli/test_forge_agent_mode.py, sections 8 and 9, and the --agent help test:
  • the no-provider subprocess repro, at INFO and at DEBUG;
  • a crash named by its class;
  • redaction;
  • a stale note;
  • --seed-from;
  • MCP forge_run logging;
  • the help's event list;
  • blank mode reporting once, under the cwd and outside it.
• tests/cli/test_forge_mode_errors.py: the helper, describe_forge_failure, the boundary, the agent loop, template mode, and _run_main printing a fix.
• tests/test_litellm_banner.py: uses the installed litellm, made to fail with mock_response=Exception(...) and no network. It covers _get_litellm, ai test --json, ai setup's key check and the agent loop. It skips when a litellm prints no banner.

Gates, on the final tree.

• These pass:
  • ruff check fluid_build/ tests/
  • the CI dead-code sweep, with no new F401/F841/F811 against main in the changed files
  • black 24.10.0
  • the product guardrail, with --self-test
  • scripts/check_license_headers.py
  • lint-imports (6 kept)
  • scripts/mypy_strict.py
• detect-secrets-hook 1.5.0 over the changed files, fed through xargs, returns 0. The only baseline entry in a touched file (ai_setup.py:154) sits above the edits.
• ripwire --quality-delta on the diff applied to a pristine main worktree exits 2. Pristine main exits 2 on its own (23 dead-code rows with no change at all). This diff adds 3 short-horizon-churn rows and 2 small test-shape clones, and no complexity, nesting or params regression.

Tests in a lean .[dev,local] venv.

• Related set on the final tree: 3895 passed. It includes tests/perf/test_startup_budget.py and tests/cli/test_lazy_core_commands.py.
• Full suite before the main merge: 34597 passed, 4 failed.
  • The two tests/providers/test_duckdb_sandbox_workspace_root.py tests fail the same way on main.
  • The other two pass alone (xdist flakes): test_dotenv_pairing_matrix::test_an_operator_exported_aws_endpoint_is_used and test_iac_databricks_metadata_fake_e2e::test_certified_then_deprecated....

Live runs through the fluid CLI, on the final tree.

• Setup: a hermetic HOME, a null keyring, a dead proxy so the local Ollama is not found, and a PATH without coding-agent CLIs.
• Every stdout line was JSON in all of these:
  • forge --agent --context x: forge.start, forge.error (copilot_no_llm_provider), forge.done. No traceback at INFO; one at DEBUG.
  • --seed-from on a missing file: forge_seed_failed.
  • --llm-provider anthropic without a key: copilot_missing_llm_api_key.
  • --blank: one forge.contract_written.
  • --template analytics: one forge.contract_written.
  • ai test --json with a dummy key: one JSON document, with error.code copilot_litellm_request_failed.

Reviews (/ship gate): passed after 1 fix round.

• Security review: no findings. An Opus judge confirmed that, and a re-review of the fix round found nothing. The one LOW fixed here: a newline in a message could forge a server-log line.
• World-class inspection (diff, delivery review): CONCERNS with one P2, the --agent help missing forge.error. It is fixed, and a re-check confirmed all five fixes. UX lane: PASS.
• Defect scan (diff): no Critical or High. The one Medium ("a stop without an exception sends no forge.error") is NOT_A_BUG: the two return 1 sites are untouched, emit what they did on main, and the behaviour is documented and pinned by a test. LOWs fixed here: the summarizer now quiets litellm, and the new tests clean up their note.

Follow-ups, not in this PR:

• Send forge.error for stops without an exception: template not found, a rejected composition, an existing blank contract.
• Send forge.error for usage errors.
• Typed mcp_sampling_failed and litellm messages embed (redacted) exception text that now reaches the IDE's model.
• Pre-existing redactor gaps: Groq gsk_, ?key=, SAS sig=, a bare AWS secret. Keyring keys are never register_secreted.
• _run_forge_inproc treats any captured JSON line with an event key as a real event.
• Concurrent MCP forge_run calls share the global stdout redirect.
• Team memory prints twice on stderr.
• litellm fetches its model-cost map at import: about 7 s with no network.

Screenshots / Output

fluid forge --agent --context x -d prod with no provider. Before (origin/main):

stdout:
{"event": "forge.start", ..., "mode": "ai", ...}
{"event": "forge.done", ..., "exit_code": 1}
stderr:
AI Copilot minimal flow failed
Traceback (most recent call last):
  File ".../fluid_build/cli/_template_mode.py", line 458, in _create_project_minimal
  ...
fluid_build.llm.providers.CopilotGenerationError: copilot_no_llm_provider
❌ No LLM provider configured. Run 'fluid ai setup' or set an API key env var.
• Pass --llm-provider, for example anthropic, openai, gemini or ollama
...

After:

stdout:
{"event": "forge.start", ..., "mode": "ai", ...}
{"event": "forge.error", ..., "code": "copilot_no_llm_provider", "message": "No LLM provider configured. Run 'fluid ai setup' or set an API key env var.", "suggestions": [...], "docs_url": "https://agenticstiger.github.io/forge_docs/advanced/production-troubleshooting.html"}
{"event": "forge.done", ..., "exit_code": 1}
stderr:
❌ No LLM provider configured. Run 'fluid ai setup' or set an API key env var.
• Pass --llm-provider, for example anthropic, openai, gemini or ollama
...

fluid ai test --json on a failed call. Before: four banner lines (Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new ...) came ahead of {. After: the JSON report alone.